# Using PatternSequence and __ / Repeated[]

I'm trying to scrape this Wikipedia page ('Deaths in 2019- July', I'm writing this on the 14th.). I am failing miserably and I'm starting to think that it is due to some misconception on my part on the use of patterns.

Namely, the use of __ vs .. and PatternSequence.

I use Import[<URL>, "XML"] to import the page. I use pattn0 (see below) to identify the positions pos0 = Position[<XML>, pattn0] and pos1 = MapAt[# + 1 &, pos0, {All, -1}].

pos and pos1 are used to identify the positions of the desired entries.

I'm using src = Apply[Part[<XML>, ##] &, Riffle[pos0, pos1], 1] in all pattern matching attempts as the source over which I try to extract (or match against) parts I want.

(I provide a link to the src used, at the end of the question)

## Snippet of desired entries

I will report below a random pair of entries I am interested in:

XMLElement["h3", {}, {XMLElement["span", {"class" -> "mw-headline", "id" -> "14"}, {"14"}]}],

XMLElement["ul",{},{
XMLElement["li",{},{
XMLElement["sup",{"id"->"cite_ref-1","class"->"reference"},{XMLElement["a",{"href"->"#cite_note-1"},{"[1]"}]}]
}
],
XMLElement["li",{},{
XMLElement["a",{"href"->"/wiki/Nereo_Laroni","title"->"Nereo Laroni"},{"Nereo Laroni"}],", 76, Italian politician,",
XMLElement["a",{"href"->"/wiki/Mayor_of_Venice","title"->"Mayor of Venice"},{"Mayor of Venice"}],"(1985\[Dash]1987) and",
XMLElement["a",{"href"->"/wiki/Member_of_the_European_Parliament","title"->"Member of the European Parliament"},{"MEP"}],"(1989\[Dash]1994), complications from heart surgery.",
XMLElement["sup",{"id"->"cite_ref-2","class"->"reference"},{XMLElement["a",{"href"->"#cite_note-2"},{"[2]"}]}]
}
],
XMLElement["li",{},{
XMLElement["a",{"href"->"/wiki/Parliament_of_Fiji","title"->"Parliament of Fiji"},{"MP"}],"(1999\[Dash]2001, 2006) and President of the",
XMLElement["a",{"href"->"/wiki/Fiji_Labour_Party","title"->"Fiji Labour Party"},{"Labour Party"}],"(since 2015).",
XMLElement["sup",{"id"->"cite_ref-3","class"->"reference"},{XMLElement["a",{"href"->"#cite_note-3"},{"[3]"}]}]
}
]
}
]


## Patterns I-am trying to-use

I am using the following patterns to match the entries I am interested in:

pattn0 = XMLElement["h3", _, {
XMLElement["span", {"class" -> "mw-headline", "id" -> id_}, {idnm_}]
}
]


(pattn0 matches against the day numbers reported in the header for each day)

and

pattn1 = XMLElement["a", {"href" -> href_, Longest[othr___], "title" -> tit_, last___}, {nm_}];
pattn2 = PatternSequence[pattn1, Longest[cmnt___]];

• pattn2 is used to match against entries like XMLElement["a", {__}, {_}] which are the data entries I'm (mostly) interested in.
• In pattn2 I am using Longest[othr___] because there are some entries where, between "href" and "title", there interjects some other minor entry (some XMLElement[__]).
• Also, last___ is there to match against any remaining entries I might have missed.
• Lastly, Longest[cmnts___] is used to match against entries (usually, as far as I can tell, a single one) after XMLElement["a", __].

pattn3 and pattn4 match against reference entries:

pattn3 = XMLElement["a", {"href" -> href2_}, {refn_}];
pattn4 = XMLElement["sup", {"id" -> id_, "class" -> "reference", {pattn3}];


(pattn4 is used to match against XMLElement["sup", {__, {__}] which are the entries related to references and their links)

pattn5 combines all the patterns into one (unfortunately non-performing) pattern:

pattn5 = PatternSequence[pattn0, x : XMLElement["ul", _, {XMLElement["li", _, {pattn2 .., pattn4}] ..}]];


## Things that work

• Retrieving day numbers Evaluating Cases[src, pattn0 :> idnm] works! and returns the day numbers

{14,13,12,11,10,9,8,7,6,5,4,3,2,1}


as expected.

• Retrieving footnotes/references

Both pattn3 and pattn4 work. Here, evaluating Cases[src, pattn4, Infinity] works (first five results):

{
XMLElement[sup,{id->cite_ref-1,class->reference},{XMLElement[a,{href->#cite_note-1},{[1]}]}],
XMLElement[sup,{id->cite_ref-2,class->reference},{XMLElement[a,{href->#cite_note-2},{[2]}]}],
XMLElement[sup,{id->cite_ref-3,class->reference},{XMLElement[a,{href->#cite_note-3},{[3]}]}],
XMLElement[sup,{id->cite_ref-4,class->reference},{XMLElement[a,{href->#cite_note-4},{[4]}]}],
XMLElement[sup,{id->cite_ref-5,class->reference},{XMLElement[a,{href->#cite_note-5},{[5]}]}],...}


## Things that partially work

• pattn2

In using pattn2 I get partially matched results; namely, the part related to Longest[cmnt___] does not get matched, ever!. Essentially what works seems to be the part related to pattn1.

Cases[src, pattn2, Infinity]


evaluates to the partially correct result (only the first five entries shown here to avoid clutter)

{
XMLElement[a,{href->/wiki/Nereo_Laroni,title->Nereo Laroni},{Nereo Laroni}],
XMLElement[a,{href->/wiki/Mayor_of_Venice,title->Mayor of Venice},{Mayor of Venice}], ...}


## Things that should have(?) worked but don't!

• Combined pattern pattn5

Evaluating Cases[src, pattn5, Infinity] evaluates to {}. Nothing gets matched!

Are all welcome.

Thanks!

## Appendix

I provide the src used throughout the question (pastebin). Run Uncompress on the string to retrieve the XML elements.

• related to pastebin: 1. click on the link 2. when you are redirected, copy the text (place your cursor in the body of the text, select all the text and copy selection) 3. in a notebook type Uncompress[" then paste in the contents of the clipboard and continue typing without moving the cursor "] 4. evaluate – user42582 Jul 15 '19 at 22:03

## Things that partially work

I got the input from the web (I was unable to use pastebin successfully).

xml = Import["https://en.wikipedia.org/wiki/Deaths_in_2019#July", "XML"];

pattn0 = XMLElement[
"h3", _, {XMLElement["span",
{"class" -> "mw-headline", "id" -> id_}, {idnm_}]}];

pos0 = Position[xml, pattn0];

pos1 = MapAt[# + 1 &, pos0, {All, -1}];

src = Apply[Part[xml, ##] &, Riffle[pos0, pos1], 1];


src changes with time so mine may not be identical to the data in pastebin.

Below are some example parts of src that serve as your data entries.

Column[src[[8, 3]][[1 ;; 2]], Dividers -> All]


Note carefully that it looks like you have multiple elements following the XMLElement. However if you check the Head you will find that it is a single string.

src[[8, 3, 3, 3, 2]]
(* ", 91, Vietnamese mathematician." *)

(* String *)


## Patterns that match XML

You had set pattn1 to

pattn1 = XMLElement["a", {"href" -> href_, Longest[othr___],
"title" -> tit_, last___}, {nm_}]


When applied to src I got 576 entries

pattn1List = Cases[src, pattn1, Infinity];


Below is a sample of the result

Column[pattn1List[[1 ;; 3]], Dividers -> All]


## Patterns that match XML and comments

You had wanted to match both the XML and comments that might follow. You tried:

pattn2 = PatternSequence[pattn1, Longest[cmnt___]]


But when it is applied it to src you didn't get any comments. In fact, it was identical to only using pattn1.

pattn2List = Cases[src, pattn2, Infinity];

SameQ[pattn1List, pattn2List]
(* True *)


## PatternSequence

The documentation indicates that PatternSequence should not be used as a standalone pattern. pattn2 violates that principle.

I will show two attempts that produce a result, however neither is the desired result.

Failure 1

pattn2 = {___, PatternSequence[pattn1, cmnt_String], ___}
pattn2List = Cases[src, pattn2, Infinity];


This produces a result but rather than individual entries it returns the entire list with interspersed XMLElements and comments. Below is an example.

pattn2List[[4]]

(* {XMLElement[
"a", {"href" -> "/wiki/Giuseppe_Merlo",
"title" -> "Giuseppe Merlo"}, {"Giuseppe Merlo"}],
", 91, Italian tennis player.",
XMLElement[
"sup", {"id" -> "cite_ref-4",
"class" -> "reference"}, {XMLElement[
"a", {"href" -> "#cite_note-4"}, {"[4]"}]}]} *)


Failure 2

In order to extract both the XMLElement and its associated comment I tried.

pattn2 = {___, group : PatternSequence[pattn1, cmnt_String], ___}
pattn2List = Cases[src, pattn2 :> group, Infinity];


This succeeded in the sense that an XMLElement and its associated comment were returned. However only the first element from a multi-element list was returned, the others were missing.

Below is a sample

Column[pattn2List[[57 ;; 62]], Dividers -> All]


## XMElement and comment

Hopefully this answer has been helpful in achieving a better understanding of PatternSequence.

I could stop here but would also like to present a method for retrieving XMLElement that match pattn1 along its associated comment. Note, not all XMLElements that match pattn1 are followed by comments.

The strategy is to make a list of positions (posPattn1)that match pattn1.

Also make a list of positions (posCmnt) that are strings.

Iterate over posPattn1, increment the position by one and determine if it is a member of posCmnt (i.e., a string).

If it is, output both, otherwise only output the XMLElement.

posPattn1 = Position[src, pattn1];
posCmnt = Position[src, _String];

pattn2List  = Map[
Module[
{
pos1 = #,
pos2 = #,
pat1,
str2
},
pat1 = Extract[src, pos1];
pos2[[-1]] = pos2[[-1]] + 1;
str2 = If[
MemberQ[posCmnt, pos2],
Extract[src, pos2],
Nothing
];
{pat1, str2}
] &,
posPattn1
];