Using PatternSequence and __ / Repeated[]

Question

I'm trying to scrape this Wikipedia page ('Deaths in 2019- July', I'm writing this on the 14th.). I am failing miserably and I'm starting to think that it is due to some misconception on my part on the use of patterns.

Namely, the use of __ vs .. and PatternSequence.

Please help.

I use Import[<URL>, "XML"] to import the page. I use pattn0 (see below) to identify the positions pos0 = Position[<XML>, pattn0] and pos1 = MapAt[# + 1 &, pos0, {All, -1}].

pos and pos1 are used to identify the positions of the desired entries.

I'm using src = Apply[Part[<XML>, ##] &, Riffle[pos0, pos1], 1] in all pattern matching attempts as the source over which I try to extract (or match against) parts I want.

(I provide a link to the src used, at the end of the question)

Snippet of desired entries

I will report below a random pair of entries I am interested in:

XMLElement["h3", {}, {XMLElement["span", {"class" -> "mw-headline", "id" -> "14"}, {"14"}]}],

XMLElement["ul",{},{
  XMLElement["li",{},{
    XMLElement["a",{"href"->"/wiki/Hussain_Muhammad_Ershad","title"->"Hussain Muhammad Ershad"},{"Hussain Muhammad Ershad"}],", 89, Bangladeshi military officer and politician,",
    XMLElement["a",{"href"->"/wiki/Chief_of_Army_Staff_(Bangladesh)","title"->"Chief of Army Staff (Bangladesh)"},{"Chief of Army Staff"}],"(1978\[Dash]1986) and",
    XMLElement["a",{"href"->"/wiki/President_of_Bangladesh","title"->"President of Bangladesh"},{"President"}],"(1983\[Dash]1990), myelodysplastic syndrome.",
    XMLElement["sup",{"id"->"cite_ref-1","class"->"reference"},{XMLElement["a",{"href"->"#cite_note-1"},{"[1]"}]}]
   }
  ],
XMLElement["li",{},{
    XMLElement["a",{"href"->"/wiki/Nereo_Laroni","title"->"Nereo Laroni"},{"Nereo Laroni"}],", 76, Italian politician,",
    XMLElement["a",{"href"->"/wiki/Mayor_of_Venice","title"->"Mayor of Venice"},{"Mayor of Venice"}],"(1985\[Dash]1987) and",
    XMLElement["a",{"href"->"/wiki/Member_of_the_European_Parliament","title"->"Member of the European Parliament"},{"MEP"}],"(1989\[Dash]1994), complications from heart surgery.",
    XMLElement["sup",{"id"->"cite_ref-2","class"->"reference"},{XMLElement["a",{"href"->"#cite_note-2"},{"[2]"}]}]
   }
  ],
XMLElement["li",{},{
    XMLElement["a",{"href"->"/wiki/Lavenia_Padarath","title"->"Lavenia Padarath"},{"Lavenia Padarath"}],", 74, Fijian politician,",
    XMLElement["a",{"href"->"/wiki/Parliament_of_Fiji","title"->"Parliament of Fiji"},{"MP"}],"(1999\[Dash]2001, 2006) and President of the",
    XMLElement["a",{"href"->"/wiki/Fiji_Labour_Party","title"->"Fiji Labour Party"},{"Labour Party"}],"(since 2015).",
    XMLElement["sup",{"id"->"cite_ref-3","class"->"reference"},{XMLElement["a",{"href"->"#cite_note-3"},{"[3]"}]}]
   }
  ]
 }
]

Patterns I-am trying to-use

I am using the following patterns to match the entries I am interested in:

pattn0 = XMLElement["h3", _, {
   XMLElement["span", {"class" -> "mw-headline", "id" -> id_}, {idnm_}]
  }
 ]

(pattn0 matches against the day numbers reported in the header for each day)

and

pattn1 = XMLElement["a", {"href" -> href_, Longest[othr___], "title" -> tit_, last___}, {nm_}];
pattn2 = PatternSequence[pattn1, Longest[cmnt___]];

• pattn2 is used to match against entries like XMLElement["a", {__}, {_}] which are the data entries I'm (mostly) interested in.
• In pattn2 I am using Longest[othr___] because there are some entries where, between "href" and "title", there interjects some other minor entry (some XMLElement[__]).
• Also, last___ is there to match against any remaining entries I might have missed.
• Lastly, Longest[cmnts___] is used to match against entries (usually, as far as I can tell, a single one) after XMLElement["a", __].

pattn3 and pattn4 match against reference entries:

pattn3 = XMLElement["a", {"href" -> href2_}, {refn_}];
pattn4 = XMLElement["sup", {"id" -> id_, "class" -> "reference", {pattn3}];

(pattn4 is used to match against XMLElement["sup", {__, {__}] which are the entries related to references and their links)

pattn5 combines all the patterns into one (unfortunately non-performing) pattern:

pattn5 = PatternSequence[pattn0, x : XMLElement["ul", _, {XMLElement["li", _, {pattn2 .., pattn4}] ..}]];

Things that work

• Retrieving day numbers Evaluating Cases[src, pattn0 :> idnm] works! and returns the day numbers

  {14,13,12,11,10,9,8,7,6,5,4,3,2,1}
  

  as expected.

• Retrieving footnotes/references

  Both pattn3 and pattn4 work. Here, evaluating Cases[src, pattn4, Infinity] works (first five results):

  { 
      XMLElement[sup,{id->cite_ref-1,class->reference},{XMLElement[a,{href->#cite_note-1},{[1]}]}],
      XMLElement[sup,{id->cite_ref-2,class->reference},{XMLElement[a,{href->#cite_note-2},{[2]}]}],
      XMLElement[sup,{id->cite_ref-3,class->reference},{XMLElement[a,{href->#cite_note-3},{[3]}]}],
      XMLElement[sup,{id->cite_ref-4,class->reference},{XMLElement[a,{href->#cite_note-4},{[4]}]}],
      XMLElement[sup,{id->cite_ref-5,class->reference},{XMLElement[a,{href->#cite_note-5},{[5]}]}],...}
  

Things that partially work

• pattn2

  In using pattn2 I get partially matched results; namely, the part related to Longest[cmnt___] does not get matched, ever!. Essentially what works seems to be the part related to pattn1.

  Cases[src, pattn2, Infinity]
  

  evaluates to the partially correct result (only the first five entries shown here to avoid clutter)

  {
  XMLElement[a,{href->/wiki/Hussain_Muhammad_Ershad,title->Hussain Muhammad Ershad},{Hussain Muhammad Ershad}],
  XMLElement[a,{href->/wiki/Chief_of_Army_Staff_(Bangladesh),title->Chief of Army Staff (Bangladesh)},{Chief of Army Staff}],
  XMLElement[a,{href->/wiki/President_of_Bangladesh,title->President of Bangladesh},{President}],
  XMLElement[a,{href->/wiki/Nereo_Laroni,title->Nereo Laroni},{Nereo Laroni}],
  XMLElement[a,{href->/wiki/Mayor_of_Venice,title->Mayor of Venice},{Mayor of Venice}], ...}
  

Things that should have(?) worked but don't!

• Combined pattern pattn5

  Evaluating Cases[src, pattn5, Infinity] evaluates to {}. Nothing gets matched!

Suggestions, Nudges, Tips on how to get a decent match

Are all welcome.

Thanks!

---

Appendix

I provide the src used throughout the question (pastebin). Run Uncompress on the string to retrieve the XML elements.

Answer 1

This answer will be about the section entitled

Things that partially work

I got the input from the web (I was unable to use pastebin successfully).

xml = Import["https://en.wikipedia.org/wiki/Deaths_in_2019#July", "XML"];

pattn0 = XMLElement[
  "h3", _, {XMLElement["span",
     {"class" -> "mw-headline", "id" -> id_}, {idnm_}]}];

pos0 = Position[xml, pattn0];

pos1 = MapAt[# + 1 &, pos0, {All, -1}];

src = Apply[Part[xml, ##] &, Riffle[pos0, pos1], 1];

src changes with time so mine may not be identical to the data in pastebin.

Below are some example parts of src that serve as your data entries.

Column[src[[8, 3]][[1 ;; 2]], Dividers -> All]

Note carefully that it looks like you have multiple elements following the XMLElement. However if you check the Head you will find that it is a single string.

src[[8, 3, 3, 3, 2]]
(* ", 91, Vietnamese mathematician." *)

Head[src[[8, 3, 3, 3, 2]]]
(* String *)

Patterns that match XML

You had set pattn1 to

pattn1 = XMLElement["a", {"href" -> href_, Longest[othr___],
  "title" -> tit_, last___}, {nm_}]

When applied to src I got 576 entries

pattn1List = Cases[src, pattn1, Infinity];

Below is a sample of the result

Column[pattn1List[[1 ;; 3]], Dividers -> All]

Patterns that match XML and comments

You had wanted to match both the XML and comments that might follow. You tried:

pattn2 = PatternSequence[pattn1, Longest[cmnt___]]

But when it is applied it to src you didn't get any comments. In fact, it was identical to only using pattn1.

pattn2List = Cases[src, pattn2, Infinity];

SameQ[pattn1List, pattn2List]
(* True *)

PatternSequence

The documentation indicates that PatternSequence should not be used as a standalone pattern. pattn2 violates that principle.

I will show two attempts that produce a result, however neither is the desired result.

Failure 1

pattn2 = {___, PatternSequence[pattn1, cmnt_String], ___}
pattn2List = Cases[src, pattn2, Infinity];

This produces a result but rather than individual entries it returns the entire list with interspersed XMLElements and comments. Below is an example.

pattn2List[[4]]

  (* {XMLElement[
  "a", {"href" -> "/wiki/Giuseppe_Merlo", 
   "title" -> "Giuseppe Merlo"}, {"Giuseppe Merlo"}],
   ", 91, Italian tennis player.", 
 XMLElement[
  "sup", {"id" -> "cite_ref-4", 
   "class" -> "reference"}, {XMLElement[
    "a", {"href" -> "#cite_note-4"}, {"[4]"}]}]} *)

Failure 2

In order to extract both the XMLElement and its associated comment I tried.

pattn2 = {___, group : PatternSequence[pattn1, cmnt_String], ___}
pattn2List = Cases[src, pattn2 :> group, Infinity];

This succeeded in the sense that an XMLElement and its associated comment were returned. However only the first element from a multi-element list was returned, the others were missing.

Below is a sample

Column[pattn2List[[57 ;; 62]], Dividers -> All]

XMElement and comment

Hopefully this answer has been helpful in achieving a better understanding of PatternSequence.

I could stop here but would also like to present a method for retrieving XMLElement that match pattn1 along its associated comment. Note, not all XMLElements that match pattn1 are followed by comments.

The strategy is to make a list of positions (posPattn1)that match pattn1.

Also make a list of positions (posCmnt) that are strings.

Iterate over posPattn1, increment the position by one and determine if it is a member of posCmnt (i.e., a string).

If it is, output both, otherwise only output the XMLElement.

posPattn1 = Position[src, pattn1];
posCmnt = Position[src, _String];

pattn2List  = Map[
   Module[
     {
      pos1 = #,
      pos2 = #,
      pat1,
      str2
      },
     pat1 = Extract[src, pos1];
     pos2[[-1]] = pos2[[-1]] + 1;
     str2 = If[
       MemberQ[posCmnt, pos2],
       Extract[src, pos2],
       Nothing
       ];
     {pat1, str2}
     ] &,
   posPattn1
   ];